Assessing the structural quality of conversations

ABSTRACT

Input of a conversation is received. The conversation includes at least a first user. An utterance of the conversation is analyzed to identify a dialog act attribute, an emotion attribute, and a tone attribute. The dialog act attribute, emotion attribute, and tone attribute are annotated to the utterance of the conversation. The conversation is validated based on the annotated attributes compared with a threshold. The annotated conversation and the validation of the conversation are stored.

BACKGROUND

The present invention relates generally to the field of conversation,and more particularly to assessing the structural quality ofconversations.

Customer satisfaction is an important metric to companies as happycustomers drive profits. A popular mechanism for assisting customers andkeeping customers happy is the customer service center, also known as acustomer contact center. A customer service center assists the customerby providing a company contact (human) or an automated system (or bot,i.e., an application that performs an automated task) for orderingproducts, providing information, resolving concerns, and the like.Conversations between customers and the company representative (human orbot) are monitored for quality control as a means of improvingcommunication and maintaining high customer satisfaction.

SUMMARY OF THE INVENTION

Embodiments of the present invention include an approach for assessingthe structural quality of conversations. In one embodiment, input of aconversation is received. The conversation includes at least a firstuser. An utterance of the conversation is analyzed to identify a dialogact attribute, an emotion attribute, and a tone attribute. The dialogact attribute, emotion attribute, and tone attribute are annotated tothe utterance of the conversation. The conversation is validated basedon the annotated attributes compared with a threshold. The annotatedconversation and the validation of the conversation are stored.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a functional block diagram of a computing environment, inaccordance with an embodiment of the present invention;

FIG. 2 depicts a flowchart of a program for assessing the structuralquality of a conversation in real time, in accordance with an embodimentof the present invention;

FIG. 3 depicts a flowchart of a program for assessing the structuralquality of a plurality of conversations, in accordance with anembodiment of the present invention; and

FIG. 4 depicts a block diagram of components of the computingenvironment of FIG. 1, in accordance with an embodiment of the presentinvention.

DETAILED DESCRIPTION

Embodiments of the present invention provide for assessing thestructural quality of conversations. A customer calling a service centerto complain about an issue regarding a purchased product starts outsomewhat irritated at the beginning of the call. Depending on thedisposition and effectiveness of the company representative, the callcan result in a happy customer or a dissatisfied customer. How well thecompany representative did in satisfying the customer is sometimesmeasured after the fact via a survey done at the completion of the call.If the customer was dissatisfied, the after the fact survey does nothelp resolve the issue in real time.

Current methods of conversation analysis are derived from thesociolinguistic field. Structural models are presented to analyzeconversations. However, the current models do not provide computationmodels for automatically computing the quality metric of a conversation.Current methods do not generate a multi-dimensional quality metric forassessing the structural quality of conversations using computationmodels and for ranking the conversations based on user adjustedpreference parameters for the purpose of gaining insights to improvecontact center management.

Embodiments of the present invention recognize that there is an approachfor assessing the structural quality of conversations. In an embodiment,various types of annotations such as the tone, emotion, and dialog actfor each utterance in a conversation is determined and annotated to arecord of the conversation in real time (i.e., as the conversation ison-going). The annotated conversation is analyzed against best practicepatterns and suggested dialog acts are provided to the companyrepresentative to better satisfy the customer.

The present invention will now be described in detail with reference tothe Figures.

FIG. 1 is a functional block diagram illustrating a computingenvironment, generally designated 100, in accordance with one embodimentof the present invention. FIG. 1 provides only an illustration of oneimplementation and does not imply any limitations with regard to thesystems and environments in which different embodiments may beimplemented. Many modifications to the depicted embodiment may be madeby those skilled in the art without departing from the scope of theinvention as recited by the claims.

In an embodiment, computing environment 100 includes computing device120 and server device 130, connected to network 110. In exampleembodiments, computing environment 100 may include other computingdevices (not shown in FIG. 1) such as smartwatches, cell phones,smartphones, wearable technology, phablets, tablet computers, laptopcomputers, desktop computers, other computer servers or any othercomputer system known in the art, interconnected to computing device 120and server device 130, over network 110.

In an embodiment of the present invention, computing device 120 andserver device 130 connect to network 110, which enables computing device120 and server device 130 to access other computing devices and/or datanot directly stored on computing device 120 and server device 130.Network 110 may be, for example, a short-range, low power wirelessconnection, a local area network (LAN), a telecommunications network, awide area network (WAN) such as the Internet, or any combination of thethree, and include wired, wireless, or fiber optic connections. Network110 may include one or more wired and/or wireless networks that arecapable of receiving and transmitting data, voice, and/or video signals,including multimedia signals that include voice, data, and videoinformation. In general, network 110 can be any combination ofconnections and protocols that will support communications betweencomputing device 120 and server device 130, and any other computingdevices connected to network 110, in accordance with embodiments of thepresent invention. In an embodiment, data received by another computingdevice (not shown in FIG. 1) in computing environment 100 may becommunicated to computing device 120 and server device 130 via network110.

According to embodiments of the present invention, computing device 120and server device 130 may be a laptop, tablet, or netbook personalcomputer (PC), a desktop computer, a personal digital assistant (PDA), asmartphone, a phablet, a smart watch, a fitness tracker or any otherwearable technology, a smart television, a smart digital video recorder,a security camera, a smart automobile, or any other programmableelectronic device capable of communicating with any other computingdevice within computing environment 100. In an embodiment, computingenvironment 100 includes any number of computing device 120 and serverdevice 130.

In certain embodiments, computing device 120 and/or server device 130represent a computer system utilizing clustered computers and components(e.g., database server computers, application server computers, etc.)that act as a single pool of seamless resources when accessed byelements of computing environment 100. In general, computing device 120and server device 130 are representative of any electronic device orcombination of electronic devices capable of executing computer readableprogram instructions. Computing device 120 and server device 130 mayinclude components as depicted and described in further detail withrespect to FIG. 4, in accordance with embodiments of the presentinvention.

According to an embodiment of the present invention, computing device120 includes user interface 121. In an embodiment, user interface 121provides an interface between a user of computing device 120, network110 and any other devices connected to network 110. User interface 121allows a user of computing device 120 to interact with the Internet andalso enables the user to receive an indicator of one or more previousviewing locations and a summary of viewing history on the Internet. Ingeneral, a user interface is the space where interactions between humansand machines occur. User interface 121 may be a graphical user interface(GUI) or a web user interface (WUI) and can display text, documents, webbrowser windows, user options, application interfaces, and instructionsfor operation, and include the information (such as graphic, text, andsound) that a program presents to a user and the control sequences theuser employs to control the program. User interface 121 may also bemobile application software that provides an interface between a user ofcomputing device 120 and network 110. Mobile application software, or an“app,” is a computer program designed to run on smartphones, phablets,tablet computers and other mobile devices.

According to embodiments of the present invention, server device 130includes dialog act detector 131, emotion analyzer 132, tone analyzer133, conversation annotator 134, best practice pattern analyzer 135,conversation flow refinement module 136, and conversation analysisprogram 137. In one embodiment, dialog act detector 131, emotionanalyzer 132, tone analyzer 133, conversation annotator 134, bestpractice pattern analyzer 135, and conversation flow refinement module136 are separate from conversation analysis program 137. In anotherembodiment, dialog act detector 131, emotion analyzer 132, tone analyzer133, conversation annotator 134, best practice pattern analyzer 135, andconversation flow refinement module 136 are part of conversationanalysis program 137. In yet another embodiment, some of dialog actdetector 131, emotion analyzer 132, tone analyzer 133, conversationannotator 134, best practice pattern analyzer 135, and conversation flowrefinement module 136 are separate from conversation analysis program137 while others are part of conversation analysis program 137.

In an embodiment, dialog act detector 131 is a program, a subprogram ofa larger program, an application, a plurality of applications, or mobileapplication software, which functions to detect the dialog act for eachutterance in a conversation. A program is a sequence of instructionswritten by a programmer to perform a specific task. According to anembodiment of the present invention, dialog act detector 131 isprogrammed by using supervised learning techniques based on known dialogact labels of example utterances. Feature engineering techniques areapplied to extract linguistic and semantic features from the utterances.Machine learning models are trained to predict the dialog act labels ofutterances using the extracted linguistic and semantic features. Machinelearning is the subfield of computer science that gives “computers theability to learn without being explicitly programmed”. Evolved from thestudy of pattern recognition and computational learning theory inartificial intelligence, machine learning explores the study andconstruction of algorithms that can learn from and make predictions ondata—such algorithms overcome following strictly static programinstructions by making data-driven predictions or decisions, throughbuilding a model from sample inputs. Machine learning is employed in arange of computing tasks where designing and programming explicitalgorithms with good performance is difficult or unfeasible; exampleapplications include email filtering, detection of network intruders ormalicious insiders working towards a data breach, optical characterrecognition (OCR), learning to rank and computer vision. Dialog actsinclude, but are not limited to: counter-greeting, opening statement,question, problem description, reply, acknowledgement, problemresolution, statement, acceptance, rejection, action-directive, apology,excuse, request, ultimatum, seeking permission, complaint, confirmpresence, exclamation, ridicule, and command.

According to an embodiment of the present invention, emotion analyzer132 is a program, a subprogram of a larger program, an application, aplurality of applications, or mobile application software, whichfunctions to analyze each utterance in a conversation and determine theemotion of each utterance in the conversation. A program is a sequenceof instructions written by a programmer to perform a specific task.According to an embodiment of the present invention, emotion analyzer132 is programmed by using supervised learning techniques based on knownemotion labels of example utterances. Feature engineering techniques areapplied to extract linguistic and semantic features from the utterances.Machine learning models are trained to predict the emotion labels ofutterances using the extracted linguistic and semantic features.Emotions include, but are not limited to: anger, disgust, sadness, fear,joy, happiness, elation, curious, surprise, trust, anticipation, shame,pity, envy, indignation, love, and kindness.

According to an embodiment of the present invention, tone analyzer 133is a program, a subprogram of a larger program, an application, aplurality of applications, or mobile application software, whichfunctions to analyze each utterance in a conversation and determine thetone of each utterance in the conversation. A program is a sequence ofinstructions written by a programmer to perform a specific task.According to an embodiment of the present invention, tone analyzer 133is programmed by using supervised learning techniques based on knowntone labels of example utterances. Feature engineering techniques areapplied to extract linguistic and semantic features from the utterances.Machine learning models are trained to predict the tone labels ofutterances using the extracted linguistic and semantic features. Tonesinclude, but are not limited to: bold, happy, supportive, clever,mysterious, disgruntled, irritated, outraged, startled, quizzical,bitter, absurd, annoyed, witty, submissive, light-hearted, jaded,formal, humorous, direct, defiant, disappointed, belligerent, concerned,nasty, and pragmatic.

According to an embodiment of the present invention, conversationannotator 134 is a program, a subprogram of a larger program, anapplication, a plurality of applications, or mobile applicationsoftware, which functions to annotate the dialog act, emotion, and tonefor each utterance in a conversation to each specific utterance in theconversation. A program is a sequence of instructions written by aprogrammer to perform a specific task. According to an embodiment of thepresent invention, conversation annotator 134 uses the results fromdialog act detector 131, emotion analyzer 132, and tone analyzer 133 toannotate each metric (i.e., dialog act or DA, emotion or E, and tone orT) to the appropriate utterance in the conversation. For example, thephrase “Hello, I have a quick question about my bill.” is annotated asfollows: “Hello, I have a quick question about my bill. [DA—greeting,question/E—none/T—pleasant].”

According to an embodiment of the present invention, best practicepattern analyzer 135 is a program, a subprogram of a larger program, anapplication, a plurality of applications, or mobile applicationsoftware, which functions to analyze an utterance of a conversation,along with the associated dialog act, emotion, and tone of theutterance, in order to determine the best response option to theutterance by determining a score for each utterance. A program is asequence of instructions written by a programmer to perform a specifictask. In an embodiment, best practice pattern analyzer 135 determinesthe “best practice” by comparing each utterance in the conversation thathas been annotated with the appropriate metrics to a known database of“best practice” annotated conversations. In the embodiment, the databaseof best practices was created by a best practice model. The bestpractice model is a machine learning model which learns from annotatedconversation samples containing representative “best practices” asdetermined by a user (e.g., trainer). In an embodiment, each utteranceis scored based on the number of violations determined by best practicepattern analyzer 135.

In a first embodiment for analyzing a complete conversation, bestpractice pattern analyzer 135 adopts computational approaches such assequence alignment algorithms. Both a complete conversation and the“best practices” are considered as a sequence of annotations includingdialog act, emotion, and tone. The system adopts edit distances such asthe Levenshtein distance to compute the similarity between aconversation and the “best practices”, and then ranks the conversationbased on the similarity scores. In addition, for a lengthy conversation,the system divides the long sequence of annotations into subsequencesaccording to certain criteria (e.g., the pre-defined window size) andcompares the local similarity score between a subsequence and the “bestpractices”. Local scores of the subsequences are aggregated to constructthe goal score of a conversation for ranking purposes. Best practicepattern analyzer 135 generates a report summarizing the violations ofconversations against the “best practices”.

In a second embodiment for analyzing an on-going conversation inreal-time, best practice pattern analyzer 135 predicts the next bestaction (i.e., the most appropriate tone or the proper dialog act) toadopt in a response. Information retrieval methods are applied to takethe annotations of an on-going conversation as a query to retrieve thenext action whose associated “best practices” is most similar to theon-going conversation. The similarity measure is based on sequencealignment algorithms. In another embodiment, a sequence-to-one model istrained by deep learning techniques based on “best practices”. The modelis applied to take the sequence of annotations of an on-goingconversation as the input sequence and predict a set of annotations forthe next response.

According to embodiments of the present invention, best practice patternanalyzer 135 allows users to configure the priority of conversationranking. In an embodiment, a weighted edit distance between aconversation and “best practices” is defined to assign additionalweights to certain types or sets of annotations. For example, a user mayweight “emotion” responses above “dialog act” and “tone”. In anotherembodiment, simple multi-dimensional sorting is used by specifying theorder for priority between “dialog act”, “emotion”, and “tone”annotation categories.

According to embodiments of the present invention, conversation flowrefinement module 136 uses inputs such as annotated conversations,ranked conversations, and conversation violations, determined by bestpractice pattern analyzer 135, to determine training for companyrepresentatives and to generated improved conversation scripts/dialogflow for bots that are acting as company representatives. In anembodiment, conversation flow refinement module 136 uses naturallanguage processing (NLP) techniques art such as dictionary-based andtopic-modeling approaches to extract linguistic and semantic patternsfrom the conversation violations. In another embodiment, conversationflow refinement module 136 uses highly ranked conversations to generateimproved conversation scripts by applying text augmentation techniquesto convert the highly ranked conversations into templates and thetemplates are used to generated improved conversation scripts.Conversation flow refinement module 136 also uses the highly rankedconversations to train a deep-learning based sequence-to-sequence modeland the beam-search methods to generate improved conversation scripts.In addition to the NLP techniques, text augmentation andsequence-to-sequence modeling, other techniques also well known in theart may be used to improve conversations.

According to embodiments of the present invention, conversation analysisprogram 137 is a program, a subprogram of a larger program, anapplication, a plurality of applications, or mobile applicationsoftware, which functions to assess the structural quality ofconversations. A program is a sequence of instructions written by aprogrammer to perform a specific task. Conversation analysis program 137may run by itself but may be dependent on system software (not shown inFIG. 1) to execute. In one embodiment, conversation analysis program 137functions as a stand-alone program residing on server device 130. Inanother embodiment, conversation analysis program 137 may work inconjunction with other programs, applications, etc., found in computingenvironment 100. In yet another embodiment, conversation analysisprogram 137 may be found on computing device 120 or on other computingdevices (not shown in FIG. 1) in computing environment 100, which areinterconnected to server device 130 via network 110.

FIG. 2 is a flowchart of workflow 200 depicting an approach forpreventing activity delays using analysis from smart computing devices.In one embodiment, the method of workflow 200 is performed byconversation analysis program 137. In an alternative embodiment, themethod of workflow 200 may be performed by any other program workingwith conversation analysis program 137. In an embodiment, a user, viauser interface 121, may invoke workflow 200 upon opening an applicationon computing device 120. In an alternative embodiment, a user may invokeworkflow 200 upon accessing conversation analysis program 137.

In an embodiment, conversation analysis program 137 receives input (step202). In other words, conversation analysis program 137 receives inputof a conversation occurring in real time. In an embodiment, theconversation is between two people (e.g., a customer and a companyrepresentative). In another embodiment, the conversation is between aperson (e.g., a customer) and a bot (e.g., an automated response systemrepresenting a company). According to an embodiment of the presentinvention, conversation analysis program 137 receives the conversationin real time as the conversation is on-going between the two parties. Inan embodiment, the conversation is separated into individual utterancesor a combination of utterances that results in a complete sentence. Inan embodiment, conversation analysis program 137 receives conversationinput between a first user of computing device 120 and a second user ofserver device 130; the conversation uses network 110 to enablecommunication between computing device 120 and server device 130. Forexample, customer “Jill” is chatting to company representative “Dan”over the Internet regarding an item “Jill” purchased from the companyrepresented by “Dan”.

In the example, a portion of the conversation proceeds as follows:

-   -   “Jill”—Can you delete this account or not?    -   “Dan”—I will refer you to our account specialist so this account        gets deleted, okay?    -   “Jill”—This does not make any sense!    -   “Dan”—I am really sorry about this inconvenience but our account        specialist will handle this for your, okay?    -   “Jill”—Fine, please put me in touch with someone that can help        me.

In an embodiment, conversation analysis program 137 receives attributes(step 204). In other words, conversation analysis program 137 receivesdialog act (DA), emotion (E), and tone (T) metric attributes from dialogact detector 131, emotion analyzer 132, and tone analyzer 133,respectively, for each utterance in the conversation. In an embodiment,conversation analysis program 137 receives the metric attributesdirectly from dialog act detector 131, emotion analyzer 132, and toneanalyzer 133. In another embodiment, conversation analysis program 137receives the metric attributes from a database stored to a memory (notshown in FIG. 1) included on server device 130. For example,conversation analysis program 137 receives the following metricattributes for the on-going conversation between “Jill” and “Dan”:question, frustration, anger, seeking permission, apologetic, fear,ridicule, confident, analytical, command, satisfied, and neutral.

In an embodiment, conversation analysis program 137 annotatesconversation (step 206). In other words, conversation analysis program137 annotates the received dialog act, emotion, and tone metricattributes to a transcript of the on-going conversation. In anembodiment, the conversation is annotated by conversation annotator 134.In another embodiment, the conversation is annotated directly byconversation analysis program 137. In an embodiment, conversationanalysis program 137 creates a transcript of the on-going conversationbetween the two parties and then annotates the received attributes tothe appropriate utterances of the conversation.

In the example, the conversation is annotated as follows:

-   -   “Jill”—Can you delete this account or not?[Dialog Act:        Question][Emotion: Anger][Tone: Frustration]    -   “Dan”—I will refer you to our account specialist so this account        gets deleted, okay?[Dialog Act: Seeking Permission][Emotion:        Fear][Tone: Apologetic]    -   “Jill”—This does not make any sense![Dialog Act:        Ridicule][Emotion: Anger][Tone: Confident/Analytical]    -   “Dan”—I am really sorry about this inconvenience but our account        specialist will handle this for your, okay?[Dialog Act: Seeking        Permission][Emotion: Fear][Tone: Apologetic]    -   “Jill”—Fine, please put me in touch with someone that can help        me. [Dialog Act: Command][Emotion: Neutral][Tone: Satisfied].

In an embodiment, conversation analysis program 137 validatesconversation (step 208). In other words, conversation analysis program137 uses best practice pattern analyzer 135 to validate the on-goingconversation based on the annotated metric attributes. In an embodiment,the annotated metric attributes are compared to a threshold emotionalresponse. The threshold is based on an assessment of whether thecustomer is satisfied or dissatisfied (e.g., happy or sad, content ordiscontented, pleased or irritated, etc.). In an embodiment between twohumans (i.e., a customer and a company representative), conversationanalysis program 137 validates the on-going conversation in real time bydetermining scores for each utterance and then suggesting appropriateresponses to the company representative as needed in an effort toimprove customer satisfaction. In an embodiment between a human customerand a bot, conversation analysis program 137 validates the pre-defineddialog/auto-generated response before the response is provided to thecustomer. In the embodiment, if conversation analysis program 137determines that the customer is dissatisfied, conversation analysisprogram 137 can transfer the on-going conversation to a human companyrepresentative in an effort to improve customer satisfaction or canreplace the pre-defined dialog/auto-generated response with a moreappropriate response before sending any response to the customer.

In another embodiment, conversation analysis program 137 usesconversation flow refinement module 136 to validate the conversation. Inthe embodiment, conversation flow refinement module 136 analyzes theviolation report determined by best practice pattern analyzer 135 anduses natural language processing techniques such as dictionary-basedapproaches and topic modeling approaches to extract linguistic andsemantic patterns from the violations. Conversation analysis program 137can then construct fine-grained guidelines from the extracted patternsto train customer service representatives to avoid the problem patternsin the future.

In an embodiment, conversation analysis program 137 determines whetherthe conversation is done (decision step 210). In other words,conversation analysis program 137 determines whether the on-goingconversation between the two parties is completed. In an embodiment(decision step 210, NO branch), conversation analysis program 137determines that the conversation is not done between the two parties;therefore, conversation analysis program 137 returns to step 202 toreceive additional input. In the embodiment (decision step 210, YESbranch), conversation analysis program 137 determines that theconversation is done between the two parties; therefore, conversationanalysis program 137 proceeds to step 212.

In an embodiment, conversation analysis program 137 stores conversation(step 212). In other words, conversation analysis program 137 stores theannotated conversation and the results from best practice patternanalyzer 135 to a memory. In an embodiment, the annotated conversationand the results from best practice pattern analyzer 135 for the analysisof the on-going conversation are stored to a database found in a memory(not shown in FIG. 1) on server device 130. For example, the entireannotated and scored conversation between customer “Jill” and companyrepresentative “Dan”, and any recommendations made to “Dan” for properresponses to “Jill” is stored to a memory.

FIG. 3 is a flowchart of workflow 300 depicting an approach forassessing the structural quality of a plurality of conversations. In oneembodiment, the method of workflow 300 is performed by conversationanalysis program 137. In an alternative embodiment, the method ofworkflow 300 may be performed by any other program working withconversation analysis program 137. In an embodiment, a user, via userinterface 121, may invoke workflow 300 upon opening an application oncomputing device 120. In an alternative embodiment, a user may invokeworkflow 300 upon accessing conversation analysis program 137.

In an embodiment, conversation analysis program 137 retrievesconversation (step 302). In other words, conversation analysis program137 retrieves a plurality of annotated and scored conversations from adatabase stored to a memory (not shown in FIG. 1) on server device 130.In an embodiment, conversation analysis program 137 retrieves theplurality of conversations from a pre-defined time frame (e.g., from thelast week, from the last two weeks, from the last month, etc.). Inanother embodiment, conversation analysis program 137 retrieves theplurality of conversations that are selected by a user from a list ofavailable annotated conversations. In yet another embodiment,conversation analysis program 137 retrieves the plurality ofconversations that are selected by conversation analysis program 137from a list of available annotated conversations. In yet anotherembodiment, conversation analysis program 137 retrieves the plurality ofconversations that are selected by best practice pattern analyzer 135from a list of available annotated conversations. For example, all ofthe thirty annotated conversations held over the last two weeks areretrieved from memory.

In an embodiment, conversation analysis program 137 receives analysis(step 304). In other words, conversation analysis program 137 receives acompleted analysis performed by best practice pattern analyzer 135 onthe retrieved annotated conversations. In an embodiment, the analysisperformed by best practice pattern analyzer 135 determines violationsfound in each of the retrieved annotated conversations. The violationsare based on the proper dialog act, emotion, and tone that should beincluded in a response to a given dialog act received from a user. Forexample, if a received dialog act is a greeting (e.g., “Hello”), theproper dialog act to respond with is a counter-greeting (e.g., “Hi”). Animproper response to a greeting would be a sarcastic question (e.g.,“What do you want?”). In another example, if a dialog act includes aparticular emotion (e.g., “anger”, “irritation”, “frustration”, etc.),the proper dialog act to respond with would include an emotion to defusethe situation (e.g., sincerity, apologetic, etc.). In an embodiment,best practice pattern analyzer 135 determines the number and types ofviolations for each of the retrieved annotated conversations andprovides the results to conversation analysis program 137. For example,of the thirty annotated conversations analyzed, fourteen conversationswere determined to have zero to one violations, six conversations weredetermined to have two to three violations, five conversations weredetermined to have four to five violations, three conversations weredetermined to have six to seven violations, and two conversations weredetermined to have 8 or more violations.

In an embodiment, conversation analysis program 137 ranks conversation(step 306). In other words, conversation analysis program 137 ranks theannotated conversations based on the results of the analysis completedby best practice pattern analyzer 135. According to an embodiment of thepresent invention, the ranking is done from best to worst based on thenumber of violations determined by best practice pattern analyzer 135.For example, the fourteen conversations with zero to one violations areranked “1”, the six conversations with two to three violations areranked “2”, the five conversations with four to five violations areranked “3”, the three conversations with six to seven violations areranked “4”, and the two conversations with eight or more violations areranked “5”. In other embodiments, ranking is done via sequence alignmentalgorithms, a sequence-to-one model, and a weighted edit distance, whichwere previously discussed.

In an embodiment, conversation analysis program 137 sends recommendation(step 308). In other words, conversation analysis program 137 reviewsthe results of the analysis by best practice pattern analyzer 135 forlower ranked annotated conversations and sends recommendations to a userfor improving the conversations between two parties. In an embodiment,the recommendations include training for human company representativesto better respond to customers. In another embodiment, therecommendations include changes in pre-defined dialog or auto-generatedresponses for bots which are representing a company to a human customer.According to an embodiment of the present invention, the firstrecommendations are sent based on the worst ranked annotatedconversations, the second recommendations are sent based on thenext-to-worst ranked annotated conversations, and the process continuesuntil recommendations have been sent for all of the ranked annotatedconversations. For example, a first recommendation, sent to a user toimprove the worst ranked annotated conversations, includes thefollowing: “follow a greeting with a counter-greeting and not aquestion”, “respond with compassion rather than sarcasm”, and “keep apositive tone instead of a negative tone”.

In another embodiment, conversation analysis program 137 usesconversation flow refinement module 136 to generate recommendations. Inthe embodiment, conversation flow refinement module 136 applies textaugmentation techniques to the highly ranked conversations to generateconversation templates from the highly ranked conversations which areused to create improved conversation scripts. In addition, the highlyranked conversations can be used to train a deep-learning basedsequence-to-sequence model. A beam search method can then be applied tothe model to generate improved scripts.

In an embodiment, conversation analysis program 137 stores information(step 310). In other words, conversation analysis program 137 stores theretrieved annotated conversations as a grouping, the analysis receivedfrom best practice pattern analyzer 135, the ranking of the annotatedconversations in the grouping, and the sent recommendations to adatabase in a memory. According to an embodiment of the presentinvention, the memory (not shown in FIG. 1) is included on server device130. According to another embodiment of the present invention, thememory (not shown in FIG. 1) is included on computing device 120 orincluded on any other computing device (not shown in FIG. 1) accessibleby server device 130 via network 110. In an embodiment, conversationanalysis program 137 stores the information to a database in a memoryincluded on server device 130. For example, the information regardingthe thirty annotated and scored conversations is stored to a database ina memory on a company server.

FIG. 4 depicts computer system 400, which is an example of a system thatincludes conversation analysis program 137. Computer system 400 includesprocessors 401, cache 403, memory 402, persistent storage 405,communications unit 407, input/output (I/O) interface(s) 406 andcommunications fabric 404. Communications fabric 404 providescommunications between cache 403, memory 402, persistent storage 405,communications unit 407, and input/output (I/O) interface(s) 406.Communications fabric 404 can be implemented with any architecturedesigned for passing data and/or control information between processors(such as microprocessors, communications and network processors, etc.),system memory, peripheral devices, and any other hardware componentswithin a system. For example, communications fabric 404 can beimplemented with one or more buses or a crossbar switch.

Memory 402 and persistent storage 405 are computer readable storagemedia. In this embodiment, memory 402 includes random access memory(RAM). In general, memory 402 can include any suitable volatile ornon-volatile computer readable storage media. Cache 403 is a fast memorythat enhances the performance of processors 401 by holding recentlyaccessed data, and data near recently accessed data, from memory 402.

Program instructions and data used to practice embodiments of thepresent invention may be stored in persistent storage 405 and in memory402 for execution by one or more of the respective processors 401 viacache 403. In an embodiment, persistent storage 405 includes a magnetichard disk drive. Alternatively, or in addition to a magnetic hard diskdrive, persistent storage 405 can include a solid state hard drive, asemiconductor storage device, read-only memory (ROM), erasableprogrammable read-only memory (EPROM), flash memory, or any othercomputer readable storage media that is capable of storing programinstructions or digital information.

The media used by persistent storage 405 may also be removable. Forexample, a removable hard drive may be used for persistent storage 405.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer readable storage medium that is also part of persistent storage405.

Communications unit 407, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 407 includes one or more network interface cards.Communications unit 407 may provide communications through the use ofeither or both physical and wireless communications links. Programinstructions and data used to practice embodiments of the presentinvention may be downloaded to persistent storage 405 throughcommunications unit 407.

I/O interface(s) 406 allows for input and output of data with otherdevices that may be connected to each computer system. For example, I/Ointerface 406 may provide a connection to external devices 408 such as akeyboard, keypad, a touchscreen, and/or some other suitable inputdevice. External devices 408 can also include portable computer readablestorage media such as, for example, thumb drives, portable optical ormagnetic disks, and memory cards. Software and data used to practiceembodiments of the present invention can be stored on such portablecomputer readable storage media and can be loaded onto persistentstorage 405 via I/O interface(s) 406. I/O interface(s) 406 also connectto display 409.

Display 409 provides a mechanism to display data to a user and may be,for example, a computer monitor.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

What is claimed is:
 1. A method, the method comprising: receiving, byone or more computer processors, an input of a conversation including atleast a first user; analyzing, by one or more computer processors, anutterance of the conversation to identify a dialog act attribute, anemotion attribute, and a tone attribute; annotating, by one or morecomputer processors, the utterance of the conversation with the dialogact attribute, the emotion attribute, and the tone attribute;validating, by one or more computer processors, the conversation basedon the annotated attributes in comparison with a threshold; and storing,by one or more computer processors, the annotated conversation and thevalidation of the conversation.
 2. The method of claim 1, wherein thestep of validating, by one or more computer processors, theconversation, comprises: comparing, by one or more computer processors,each annotated utterance in the conversation to a known database of bestpractice annotated conversations; scoring, by one or more computerprocessors, each annotated utterance based on a number of violations,based on the comparison of each annotated utterance to the knowndatabase of best practice annotated conversations; ranking, by one ormore computer processors, each utterance based on the scoring; andsuggesting, by one or more computer processors, a response for a seconduser to make to the first user based on each ranked utterance.
 3. Themethod of claim 1, wherein the conversation is between the first userand an automated response system, and wherein the automated responsesystem comprises an application that performs an automated task.
 4. Themethod of claim 1, further comprising: retrieving, by one or morecomputer processors, a plurality of stored annotated conversations;analyzing, by one or more computer processors, the plurality of storedannotated conversations, wherein the analysis determines a number ofviolations found in each annotated conversation; ranking, by one or morecomputer processors, each annotated conversation in the plurality ofstored annotated conversations based on the analysis; and sending, byone or more computer processors, at least one recommendation to a seconduser for improving at least one annotated conversation in the pluralityof stored annotated conversations.
 5. The method of claim 4, wherein therecommendation includes a change to a pre-defined dialog.
 6. The methodof claim 2, further comprising: determining, by one or more computerprocessors, the number of violations using an algorithm selected fromthe group consisting of: a sequence alignment algorithm, an informationretrieval method, a sequence-to-one model, and a weighted edit distance.7. A computer program product, the computer program product comprising:one or more computer readable storage media; and program instructionsstored on the one or more computer readable storage media, the programinstructions comprising: program instructions to receive an input of aconversation including at least a first user; program instructions toanalyze an utterance of the conversation to identify a dialog actattribute, an emotion attribute, and a tone attribute; programinstructions to annotate the utterance of the conversation with thedialog act attribute, the emotion attribute, and the tone attribute;program instructions to validate the conversation based on the annotatedattributes in comparison with a threshold; and program instructions tostore the annotated conversation and the validation of the conversation.8. The computer program product of claim 7, wherein the programinstructions to validate the conversation comprises: programinstructions to compare each annotated utterance in the conversation toa known database of best practice annotated conversations; programinstructions to score each annotated utterance based on a number ofviolations, based on the comparison of each annotated utterance to theknown database of best practice annotated conversations; programinstructions to rank each utterance based on the scoring; and programinstructions to suggest a response for a second user to make to thefirst user based on each ranked utterance.
 9. The computer programproduct of claim 7, wherein the conversation is between the first userand an automated response system, and wherein the automated responsesystem comprises an application that performs an automated task.
 10. Thecomputer program product of claim 7, further comprising programinstructions stored on the one or more computer readable storage media,to: retrieve a plurality of stored annotated conversations; analyze theplurality of stored annotated conversations, wherein the analysisdetermines a number of violations found in each annotated conversation;rank each annotated conversation in the plurality of stored annotatedconversations based on the analysis; and send at least onerecommendation to a second user for improving at least one annotatedconversation in the plurality of stored annotated conversations.
 11. Thecomputer program product of claim 10, wherein the recommendationincludes a change to a pre-defined dialog.
 12. The computer programproduct of claim 8, further comprising program instructions stored onthe one or more computer readable storage media, to: determine thenumber of violations using an algorithm selected from the groupconsisting of: a sequence alignment algorithm, an information retrievalmethod, a sequence-to-one model, and a weighted edit distance.
 13. Acomputer system, the computer system comprising: one or more computerprocessors; one or more computer readable storage media; and programinstructions stored on the one or more computer readable storage mediafor execution by at least one of the one or more computer processors,the program instructions comprising: program instructions to receive aninput of a conversation including at least a first user; programinstructions to analyze an utterance of the conversation to identify adialog act attribute, an emotion attribute, and a tone attribute;program instructions to annotate the utterance of the conversation withthe dialog act attribute, the emotion attribute, and the tone attribute;program instructions to validate the conversation based on the annotatedattributes in comparison with a threshold; and program instructions tostore the annotated conversation and the validation of the conversation.14. The computer system of claim 13, wherein the program instructions tovalidate the conversation comprises: program instructions to compareeach annotated utterance in the conversation to a known database of bestpractice annotated conversations; program instructions to score eachannotated utterance based on a number of violations, based on thecomparison of each annotated utterance to the known database of bestpractice annotated conversations; program instructions to rank eachutterance based on the scoring; and program instructions to suggest aresponse for a second user to make to the first user based on eachranked utterance.
 15. The computer system of claim 13, wherein theconversation is between the first user and an automated response system,and wherein the automated response system comprises an application thatperforms an automated task.
 16. The computer system of claim 13, furthercomprising program instructions stored on the one or more computerreadable storage media for execution by at least one of the one or morecomputer processors, to: retrieve a plurality of stored annotatedconversations; analyze the plurality of stored annotated conversations,wherein the analysis determines a number of violations found in eachannotated conversation; rank each annotated conversation in theplurality of stored annotated conversations based on the analysis; andsend at least one recommendation to a second user for improving at leastone annotated conversation in the plurality of stored annotatedconversations.
 17. The computer system of claim 16, wherein therecommendation includes a change to a pre-defined dialog.